1. Field of the Invention
The present invention is a hardware, apparatus, and method for measuring audience data from image stream using dynamically-configurable hardware architecture, wherein reconfigurable computational modules are used as engines per node to power the complete solution implemented in a flexible hardware architecture, and the computational modules comprise demographics classification, gaze estimation, emotion recognition, behavior analysis, and impression measurement.
2. Background of the Invention
The role of digital media for advertisement in public spaces is becoming increasingly important. The task of measuring the degree of media exposure is also deemed as very important both as a guide to the equipment installation (equipment kind, position, size, and orientation) and as a rating for the content programming. As the number of such displays is growing, measuring the viewing behavior of the audience using human intervention and other segmentation information, such as demographic information of the audience, can be very costly. Therefore, there is a need to develop a novel embedded audience measurement platform. The present invention is a hardware, apparatus, and method for measuring audience data from image stream using dynamically-configurable hardware architecture. In many of the audience measurement applications, it is also necessary to employ multiple cameras along with processing hardware where an intelligent coordination between the multiple hardware is potentially beneficial to the accurate measurement.
Previous attempts for a reconfigurable classification system, especially for the hardware implementation of reconfigurable classification, are discussed.
For example, U.S. Pat. Appl. Pub. No. 2008/0120260 of J. W. Yancey (hereinafter Yancey) disclosed a system and method for reconfigurable neural networks that utilize interconnected field programmable gate array (FPGA). In Yancey, the neural networks are highly reconfigurable in that the neural network nodes communicate with each other using dynamic packet routing rather then fixed node interconnections. Each FPGA has a packet router that provides communication connection both to the neural network nodes internal to the FPGA and to the neural network nodes in external FPGAs. A neural network controller is coupled to an FPGA neural network array and is configured to provide input, management, and control data to the FPGAs within the FPGA neural network array. The controller includes a plurality of different tasks. The neural network array is then configured to perform the selected task and operates on the input according to the selected task.
Although Yancey disclosed a reconfigurable neural network method, Yancey is entirely foreign to the classification or recognition of images or video data. In addition, Yancey is limited to only neural networks that utilize interconnected FPGAs and does not explicitly teach any other classification algorithms, whereas in the present invention, the computational module can embrace any kind of classification algorithms in the hardware architecture. In addition, any application involving combinations of classifiers is implementable by specifying a process graph for computational modules in the present invention.
Reyna-Rojas, R., D. Houzet, D. Dragomirescu, F. Carlier, and S. Ouadjaout, Object recognition system-on-chip using the support vector machines, EURASIP J. Appl. Signal Process, 2005, 1, Jan. 2005, pp. 993-1004 (hereinafter Reyna-Rojas) disclosed a design of a reconfigurable and programmable system-on-chip platform for real-time object recognition. Especially Reyna-Rojas teaches an implementation of support vector machine (SVM) classifier on a system-on-chip (SoC) platform in order to exploit the parallel nature of the SVM algorithm. The reconfiguration is performed by a C program on LEON2 SPARC processor, with dynamically-constructed vector instructions. This reconfigurable preprocessing links the parameters of the main loop of the SVM algorithm that is executed at vector processing unit at run-time. Reyna-Rojas limited classification algorithm to support vector machine algorithm. In the present invention, on the other hand, any kind of reconfigurable classification algorithms can be used.
U.S. Pat. No. 5,093,900 of H. P. Graf (hereinafter Graf) disclosed a method and system for a reconfigurable neuron using analog technique in a neural network. Graf teaches, in the reconfigurable neuron, digital input data are multiplied by programmable digital weights in a novel connection structure. The sum signal is multiplied by programmable scalar values. Scalar values can be adjusted according to the portion of input data if input data and weights have multilevels. This signal is passed through a programmable build out circuit which permits neural network reconfiguration by interconnection of one neuron to one or more other neurons.
Although Graf implemented a reconfigurable neural network with programmable weight values and flexible interconnection of the neurons, Graf is foreign to the idea of enabling multiple classifiers in the architecture. Furthermore, in the present invention, the multiple classifiers can consist of not only neural network algorithms, but also any kind of classification algorithm. In the present invention, the multiple classifiers can be combined to measure the audience information, such as behavioral and demographic attributes.
U.S. Pat. No. 5,751,913 of T-D. Chiueh, et al. (hereinafter Chiueh) disclosed a method of reconfigurable neural network including difference-square type neurons. In Chiueh, reconfigurable neural network includes a plurality of switches each having at least two conductive leads. The switches are programmed so that the data flow direction of the conductive leads of the switches can be selected reconfigurable. Additionally, each neuron called processing element (PE) of the reconfigurable neural network has a serial-in-serial-out processing that is interconnected through the plurality of switches. These PEs are selected to form a layer. The switch boxes connected to the input leads of these PEs are each configured to select their input lead to electrically connect to the other PE's output lead. Chiueh disclosed a reconfigurable neural network using reconfigurable neuron structure and allowed a programmable weight value and flexible interconnection between each PE. However, Chiueh is also foreign to the idea of enabling multiple classifiers.
Janusz A. Starzyk and Yongtao Guo, “Reconfigurable Self-Organizing NN Design Using Virtex FPGA”, Proc. Int. Conf. on Engineering of Reconfigurable Systems and Algorithms (ERSA) Las Vegas, Nev., June 2001 (hereinafter Janusz) disclosed a method and system for self-organizing neural network model with entropy-based evaluator. In Janusz's self-organization architecture, the neural network is a feed forward structure that can decide the interconnections of neurons and neuron operations, and the entropy-based evaluator (EBE) is utilized to select an operation and input selection for each neuron.
Janusz's algorithm for digital implementation of neural network based on system entropy is foreign to the idea of enabling a combination of classification algorithms as does the present invention. Janusz implemented a single neural network classification algorithm for a single data set. In addition, Janusz is entirely foreign to the idea of developing an architecture designed for audience measurement.
Bermak, A. and D. Martinez, “A reconfigurable hardware implementation of tree classifiers based on a custom chip and a CPLD for gas sensors applications,” in Proceedings of the IEEE TENCON 2004 IEEE Region 10 Conference, Vol. 1, pp. 32-35, 21-24 Nov. 2004 (hereinafter Bermak) disclosed a system and method of a reconfigurable decision tree hardware architecture that can be programmed in terms of topology and processing precision allowing the user to externally set the decision parameters. In decision tree, each node in the tree is a Threshold Logic Unit (TLU) implementing a linear discriminant and each leaf is associated to a given class. In VLSI implementation of this system, Bermak used a reconfigurable custom chip for implementation of TLU. 4 by 4 array of processing element (PE) that implements TLU internally is configured in terms of decision tree topology.
Bermak used a combination of linear discriminant for reconfigurable decision tree implementation. Bermak's implementation based on decision tree algorithm can process only one data set classification, whereas the present invention allows several classifier algorithms in the architecture so that each of the classifiers can process its own data set where the data sets among the classifiers can be different from each other.
Biasi, I., A. Boni, and A. Zorat, “A reconfigurable parallel architecture for SVM classification,” in Proceedings of IJCNN '05, Vol. 5, pp. 2867-2872, 31 Jul.-4 Aug. 2005 (hereinafter Biasi) disclosed a method of hardware implementation of SVMs on reconfigurable FPGAs. Biasi suggested a new architecture called KBLAZE, in which Biasi used an external memory to store the parameters of the classification function so that the architecture can overcome a limited memory space problem. Biasi also used a 32-bit MicroBlaze RISC processor as a control unit to improve the performance bottleneck, a parallel computation module of the dot product, and multiple copies of KBLAZE on the same FPGA to improve the classification time through parallel processing. Biasi is entirely foreign to the idea of facilitating multiple classification algorithms to measure the behavioral and demographic attributes of audience.
Mizuno, R., N. Aibe, M. Yasunaga, and I. Yoshihara, “Reconfigurable architecture for probabilistic neural network system,” in Proceedings of IEEE International Conference on Field-Programmable Technology (FPT), 2003, pp. 367-370, 15-17 Dec. 2003 (hereinafter Mizuno) disclosed a reconfigurable architecture for the probabilistic neural network (PNN), in which the preprocessing circuits can be reconfigurable as well as the PNN. For reconfigurable PNN processor using FPGA, Mizuno referenced a prior implementation, disclosed in Aibe, N.; Yasunaga, M.; Yoshihara, I.; Kim, J. H., “A probabilistic neural network hardware system using a learning-parameter parallel architecture,” Neural Networks, IJCNN '02, in Proceedings of 2002 IJCNN, Vol. 3, pp. 2270-2275, 2002 (hereinafter Aibe). Aibe proposed the sigma parallel architecture in which every processor calculates the same neuron with the different value of the smoothing parameter to overcome the limitation in the number of neurons due to the shortage of I/O pins in the chip. Mizuno noted, in the PNN system, Mizuno used three FPGA chips for the PNN processor, image preprocessing, and process control, in which reconfigurability of the preprocessor and of the PNN processor is closely related to the high system performance.
Mizuno's system is limited to the specific PNN classification algorithm implementation, unlike the present invention. In addition, Mizuno is also entirely foreign to the idea of a hardware implementation that is used for audience measurement.